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METHOD FOR DISPLAY TIME STAMPING DISCLOSURE OF INVENTION 

AND SYNCHRONIZATION OF MULTIPLE The above problems can be solved by implementing a 

VIDEO OBJECT PLANES common temporal resolution to be used for ail the local time 

base. In order to cater for the wide range of temporal 

TECHNICAL FIELD 5 granularity the local time base is partitioned into two dif- 

. , . . , , . t , rtf ,K-^ a i ,„j! n ferent parts. The first part containing the fine granularity 

Ibis invention is useful in the cooing or digital audio /. , . , u _ • \ ~, 

. r * , u .u • %™ c^w™™* temporal resolution which provide a short time base. The 

visual materials where there is a need for synchronizing v^xi F ^ ^ r 

, > i i + , „ nil rt u;-- fe second part contain a coarse granularity temporal resolution 

more than one independently encoded audio visual objects , . " * , . T J . , 

, . T . T1 „, Um . OTMnAr ,i which provide a long time base. The short time base is 

for presentation. It is especially suitable when the temporal , , i j . - i_- . , . -j i 

iui v ^ / _ 10 included at each the video object planes to provide temporal 

sampling of the audio visual material is not the same. A . J f £. ^ 

a*xiupiui e reference to the instances of the video objects. The short 

BACKGROUND ART t ^ ine ^ 3ase IS ^ en synchronized to the longtime base which 

is common to all the video object planes. It is used to 

In MPEG-1 and MPEG-2 the input video consists of synchronize all the different video object planes to a com* 

image frames sampled at regular time intervals. This repre- is mon base provided by the master clock, 

sents the finest temporal resolution of the input FIG. 1 shows According to a fist aspect of the present invention, a 

a video sequence with fixed frame rates where the image metri0 d of encoding a local time base of an audio visual 

frames are sampled at regular intervals. In the coded repre- sequence in the compressed data comprises the steps of: 

sentation of the video sequence using the MPEG-1 and instances of the audio visual sequence by tem- 
MPEG-2 standards the display order of the decoded frames 20 ^ sampling . 

is indicated by the temporal reference. This parameter detennij}in ^ local dme base of ^ said instances to be 

appears at the picture header of the bitstream syntax. The intQ 5 essed data; 
value of this parameter is incremented by one for eacn 

, , , . ' T • j • A. encoding the said local tune base in two parts, comprising 

decoded frame when examined m the display order. s r r . « 

uctoucu iioiuc »u» =/w ? j . . of a modulo time base that marks the occurrence of a set of 

In the IL263 standard it is possible to slrip frames and thus -5 ^ reference spaced evenly at a mterval on ^ 

decode a variable frame rate video sequence. However the ^ ^ bas£ . ^ & ^ basfi ^^n,. relative t0 ±& said 

sampling of frames is still fixed. Thus the temporal reference g d ^ reference; 

method used in MPEG-1 and MPEG-2 a still appropriate £ ^ ^ ^ ^ ^ ^ essed data 

and only need to be modined such that the increment u ; not ^ have el d . ^ 

in steps of 1 but m stepsof number of non- 30 P ^ 

transmitted pictures at the input frame rate. ^ rf ^ said Qf ^ ^ ^ 

Currently work is being done in the area of codmg video Accordin t0 a second of ^ invention, a method 

as separate object in multiple video object planes. This encodi ° a local time h3 £ of m audio visua i sequence in 

represents a new dunension in the decoding and synchro- compr = sed ^ comprises me st eps of: 

nkation of the respective video ^f-^^i^ obtail L g toes of L audio visJal sequence by tern- 

these different video object planes may come from several r . 

sources and can have very different frame rates. Some of the P oral sampling; 

objects may be rendered and have almost continuous tern- deterrrining the local time base of the said instances to be 

poral sampling rate. These video objects planes are com- encoded .into compressed data; 

bined into a composite image for display. Therefore some 40 encoding the said instances in one of two methods corn- 
kind of syiichronization is required for the composition. It is prising of a first method of compression without reference to 
possible that the display frame rate is different from any of any future instances and a second method of compression 
the frame rates of the video object planes. FIG. 2 shows an which refers to the future reconstructed instance; 
example of two video object planes having different and encoding the said local time base in two parts, comprising 
varying frame rates compared to one another. Even if a 45 of a modulo time base that marks the occurrence of a set of 
common frame rate can be found between the two video time reference spaced evenly at a specific interval on the 
object plane, it is not automatic that this frame rate will be local time base and a time base increment, 
the same as the output frame rate of the compositor. encoding the time base increment for the instance corn- 
Hereafter we will refer to the problem in the video pressed using the first compression method as an absolute 
domain. However the same principles of this invention can value relative to the said evenly spaced time reference; 
be extended to the audio domain as well as combinations of encoding the time base increment for the instance com- 
ftg pressed using the second compression method as a relative 
From the above it is clear that the current state of the an value to the local tune base of the previously compressed 
does not cater for the synchronization of video object planes. 55 instance using the said first method; 
The state of the art also does not provide a common temporal inserting the modulo time base into the compressed data 
reference when the different video object planes have dif- whenever the specific interval have elapsed; and 
ferent frame rates which are not multiples of one another. inserting the time base increment within the compressed 

The first problem is how to provide a common local time data of the said instances of the audio visual sequence, 

base mechanism for each video object plane. This time base 60 According to a third aspect of the invention, a method of 

will have to be able to provide very fine temporal granularity the first or the second aspect of the invention, wherein a 

and at the same time be able to cope with possibility of very plurality of compressed bicstreams containing local time 

long intervals between two successive instances of the video base information encoded therein are multiplexed, further 

object planes. comprises, for multiplexing, the steps of: 

The second problem is how to provide a mechanism for 65 syncrnroruzing the local time base of the individual corn- 
synchronizing video object planes having different frame pressed bitstream to a common time base by encoding and 
rates> inserting the time base offset in the multiplexed bitstream; 
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examining each of the compressed bitstream for the next 
compressed instance to be placed into the multiplexed 
bitstream until all the compressed bitstreams have arrived at 
the modulo time base; 

inserting a common modulo time base into the multiplex 5 
bitstream and skipping the modulo time base of the com- 
pressed bitstreams; and 

repeating the last two steps until all compressed bit- 
streams are exhausted. 

According to a fourth aspect of the invention, a method of 
the first or the second aspect of the invention, wherein a 
plurality of compressed bitstreams containing local time 
base information encoded therein are multiplexed and 
de-multiplexed, further comprises, for de-multiplexing, the ^ 
steps of: 

decoding the time base offset of the individual com- 
pressed bitstream; 

examining the multiplexed bitstream for the next com- 
pressed instance and placing the said instance into the 2 o 
appropriate compressed bitstream until a modulo time base 
is encountered in the multiplexed bitstream; 

inserting a modulo time base to each of the compressed 
bitstreams; and 

repeating the last two steps until the multiplexed bitstream 25 
is exhausted. 

According to a fifth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compressed data encoded, according to 
the first aspect of the invention, comprises the steps of: 30 

initializing the reference time base taking into account the 
time base offset; 

. incrementing the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 35 
instance; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
reference time base. 40 

According to a sixth aspect of the invention, a method of 
decoding a local time base of an audio visual sequence from 
the time base of the compressed data encoded, according to 
the second aspect of the invention, comprises the steps of 

initializing the reference time base taking into account of 45 
the time base offset; 

incrementing the reference time base by a specific interval 
for each modulo time base decoded; 

decoding the time base increment of the compressed 
instance; and 50 

determining the time base increment to be one of two 
types, absolute or relative, depending on the first or second 
compression method used in the encoding of the instance, 
respectively; 

determining the decoding time base of the said instance 55 
by adding the said decoded time base increment value to the 
reference time base if the time base is of the first type; and 

determining the decoding time base of the said instance 
by adding the said decoded time base increment value to the 
decoding time base of the previous instance encoded using 60 
the first compression method, if the time base increment is 
of the second type, 

BRIEF DESCRIPTION OF DRAWINGS 
The present invention will become more fully understood 65 
from the detailed description given below and the accom- 
panying diagrams wherein: 
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FIG. 1 illustrates the temporal sampling of the prior art 
where the frames of the video sequence is sampled at regular 
intervals. 

FIG. 2 illustrates the concept of video object planes and 
the relationship with one another. The sampling of the video 
object planes can be irregular and the sampling period can 
vary drastically, 

FIG, 3A illustrates the current invention where the tem- 
poral reference of the video objects are indicated by the 
modulo time base and the VOP time increment. Illustration 
uses I and P VOP only. 

FIG. 3B illustrates the current invention where the tem- 
poral reference of the video objects are indicated by the 
modulo time base and the VOP time increment. Illustration 
uses I, P and B VOR 

FIG. 4 illustrates an example of the ambiguity that can 
arise when the presentation order and the encoding order is 
different as a result of B- video object planes. 

FIG. 5 illustrates a solution to the ambiguity by employ- 
ing absolute and relative time bases. 

FIG. 6 illustrates the combining of two VOPs and their 
synchronization to the common time base by employing 
VOP time offsets. 

FIG. 7 illustrates the flow diagram for the encoding of the 
time base. 

FIG. 8 illustrates the flow diagram for the multiplexing of 
more than one video object planes. 

FIG. 9 illustrates the flow diagram for the de-multiplexing 
of more than one video object planes. 

FIG. 10 illustrates the flow diagram for the recovery of the 
presentation time stamp. 

FIG. U illustrates the block diagram for the operations of 
the bitstream encoder for encoding of the time base. 

FIG. 12 illustrates the block diagram for the operations of 
the bitstream decoder for decoding of the time base. 

FIG. 13 illustrates the time chart for forming a bitstream 
data. 

BEST MODE FOR CARRYING OUT THE 
INVENTION 

This invention operates by providing two forms of syn- 
chronization. The first is a short time base attached to the 
video object plane. Hereafter this time base will be referred 
to as the VOP time increment. The VOP time increment is 
the tirxiing mechanism for the video object plane with 
respect to the long rime base which, is attached to a group of 
video object planes to be decoded and composed together. 
The long time base is referred to as the modulo time base. 
The VOP time increment and the modulo time based will 
then be used jointly to determine the actual time base to use 
for the composition of the video object planes into the final 
composited sequence for display. 

In order to facilitate editing of the bitstream and combin- 
ing different video object planes from different sources into 
a new group of video object planes there is a need for a third 
component to allow a fixed amount of offset between the 
local rime base of the individual video object plane from the 
common time base. Hereafter this offset will be referred to 
as the VOP time offset. This prevents the different video 
object planes from having to synchronize at a granularity 
equivalent to the modulo time base interval. This component 
should be kept constant for each video object plane within 
the group of video object planes that are multiplexed 
together. 
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First, the modulo time base is described. 
Modulo time base represent the coarse resolution of the 
local time base. It does not have a value like the VOP time 
increment. In fact it is more of a synchronization mechanism 
to synchronize the VOP time increment to the local time 
base of the video object plane. It is placed in the encoded 
bitstream as markers to indicate that the VOP time increment 
of the video object planes that follows are to be reset and that 
the reference to the local time base are to be incremented by 
one or more units of the modulo time base interval. In FIGS. 
3 A, 3B, 4, 5 and 6, the modulo time base is shown as a series 
of zero or more "1" terminated by a "0" inserted into the 
bitstream header before the VOP time increment. The num- 
ber of "1" inserted into the bitstream depends on the number 
of units of modulo time base that has elapsed since the last 
code I or P- video object plane. At the encoder and decoder 
the modulo time base counter is incremented by one for each 
"1" encountered. Modulo time based counter is finite in 
length and therefore in a practical system the modulo time 
base is reset to zero when the maximum value is exhausted 
In a typical video sequence, the video object planes form a 
group of VOP. Therefore the modulo time base is usually 
reset at the start of this group of VOP. 
Next, the VOP time increment is described. 
The VOP time increment shall be in units that can support 
the shortest temporal sampling of the video objects, It can 
also be the native time base for the object which is rendered. 
It therefore represents the finest granularity of temporal 
resolution that is required or can be supported. 

The VOP time increment can then be indicated by a finite 
length number greater or equal to the ratio of the global time 
base interval to the local time base resolution. FIG. 3A 
shows an example of the VOP time increment for I and 
P-video object planes and the reference to the modulo time 
base. An absolute time base is used. The VOP time incre- 
ment is reset each time the modulo time base is encountered. 
FIG. 3B shows another example using. I t P and B-video 
object planes! The operation is the same with the exception 
that the modulo time base is repeated in the B-video object 40 
planes. If the modulo time base is not repeated in the 
B-video object plane then ambiguities arise due to the 
difference in decoding and presentation order. This is 
described in detail below* 

Since the VOP time increment corresponds to the presen- 
tation time base, a potential problem will arise when the 
encoding order Is different from presentation order* This 
occurs with B-video object planes. Similar to MPEG-1 and 
MPEG-2 B -pictures, B-video object planes are encoded 
after their reference I and P-video object planes even though 
they preceded the reference video objects in the presentation 
order. Since the VOP time increment is finite and is relative 
to the modulo time base, the VOP time increment is reset 
when the modulo rime base is encountered. However, the 
order of encoding for the B-video object planes have been 
delayed. FIG. 4 show the ambiguity that can arise. It is not 
possible to determine when the VOP time increment should 
be reset In fact when given a sequence of encoded events as 
in FIG. 4A T it is not possible to know which of the timing 
situations of FIGS. 4B, 4C and 4D it is trying to represent. 
This problem comes about because of the use of one modulo 
time base shared among all the different types of video 
object coupled with the difference between the encoding and 
presentation order. There is nothing that can be done to the 
encoding order since the reference information is required 
by the B-video object planes. It is also not desirable to have 
independent modulo time base for the different prediction 
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type. Therefore the solution is to code the VOP time 
increment of the B-video object planes as a relative value to 
the preceding I or P-video object planes and that the modulo 
time base applies only to the I and P-video object planes and 
not the B video object planes. This solution is depicted in 
FIG. 5. 

Next, VOP time offset is described. 

Besides the above the modulo time base are shared 
between all video object planes. This means that the syn- 
chronization of the different video objects will have a 
granularity equivalent 10 the modulo time base interval This 
is unacceptable especially in the case where video object 
planes from different groups are combined to form a new 
group of video object planes. FIG. 6 shows an example of 
two different video object planes that were encoded with two 
different local rime base that are offset from one another. 
Thus when the video object planes are multiplexed, the 
synchronization of the video objects planes are also offset. 
A finer granularity is achieved by allowing each individual 
video object plane to have a VOP time offset This means 
that only this value is changed when the video object planes 
are manipulated and multiplexed. There is no need to change 
the VOP time increment and yet it is possible to multiplex 
the different video object planes without coarse granularity 
timing differentials. FIG. 6 illustrates the use of this time 
base offsets. 

The preferred embodiment of the current invention 
involves a method of encoding the rime base for each of the 
individual video object plane bitstream, a method of multi- 
plexing different video object planes into a common rime 
base, a method of de -multiplexing the multiplexed bitstream 
into the components and a method of recovering the time 
base from the component bitstream. 

Encoding the time base is described. 

The flow diagram of the embodiment of the time base 
encoding is shown in FIG. 7. In the encoder the local time 
base is first initialized to the local start time in step 1. The 
process is passed to step 2 where the encoder detenriines the 
current value of the local time base. In step 3 the local time 
base obtained is compared to the previously encoded 
modulo time base to see if the interval has exceeded the 
modulo time base interval. If the interval has been exceeded 
the control is passed to step 4 where the required number of 
modulo time base is inserted into the bitstream. If the 
interval has not been exceeded then no special handling is 
necessary. The processing then proceeds to step 5 where the 
VOP time increment is inserted into the bitstream. The 
object is then encoded in step 6 and the inserted into the 
bitstream. The encoder then checks to detennine if there are 
more objects to be encoded in step 7. If there are more 
objects to be encoded the process resumes at step 2 where 
the local time base is obtained. If there are no more objects 
to be encoded then the process terminates. 

The following formula are used to determine the absolute 
and relative VOP time increments for I/P-video object 
pianes and B-video object planes, respectively. 
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x CTBt is the pre-deterrnined moduio time base interval, The following formula are used to determine the presen- 

tcTBo is the encoder time base start time, tation time stamp of the video objects; 
t AV77 is the absolute VOP time increment for the I or P-video 
object planes, 

x £T3u? is ^ e encoder tirae base at the start of the encoding 5 ^ar^r^cra^cm M>> 1. 2. 3, . . . ) (4) 

of the I or P video object planes, 

t RVTI is the reladve VOP time increment for the B-video W/^vr/*%m, (5) 

object planes, and ■ _ f . f 

tsTBB is the encoder time base at the start of the encoding of DT3S ~ RVrr DT3IfP 

the B video object plane. 10 w kere 

Next, multiplexing more than one video object planes is t ^ h ±Q decoding dme base marked by ^ nth decoded 

described. 4J t . . . modulo time base, 

When more than one video object planes are ^t?fcxed ^ ^ ^tsmwed modulo ^ base 

together, the muiuplexer exairunes the is me decoding time base start time, 

P^ bl «^ 15 wismedecodkgtim^ 

well as the synchronization. The operations involved is w dtsj/p a & 

depicted in FIG. 8. The VOP time offsets for each of the of the I or P video ob jec t planes 

video object planes to be multiplexed are inserted into the W, * the decoded absolute VOP time increment for the I 

bitstream in step 11. All the bitstreams of the video object or P-video object planes, 

planes to be multiplexed are then exaimned in step 12 to W, is the decoding time base at the start of the decoding 

determine if all of the planes are at their respective modulo 20 of the B video object plane, and 

time base. If they are then the processing is ^zssts to step 13 t^, is the decoded relative VOP time increment for the 

where a common modulo time base is inserted into the B-video object planes. 

multiplexed bitstream. The processing then proceeds to step Implementation of the Bitstream Encoder is described. 

14 ,where the next encoded video object is inserted into the FIG. 11 shows the block diagram for an implementation 

multiplexed bitstream. In step 15 the bitstreams of the video 25 of the bitstream encoder for encoding the modulo time base 

object planes to be multiplexed are again examined to see if and the VOP time increment. For the purpose of this 

there are more video objects to be multiplexed. If there is, illustration the example in FIG. 3B is used. Since 

then the control is passed to step 12 again. Otherwise the bi-directional prediction is used, the encoding order is 

process ends. different from the presentation order shown in FIG. 3B. The 

De-multiplexing the bitstream containing more than one 30 encoding order begins with the I- VOP followed by the 

video object planes is described, P-VOP before the B-VOP. This is illustrated in the three 

The de-multiplexing of the bitstream containing multiple paragraphs below, 

video object planes is depicted in FIG. 9. The process begins The process begins in the initializer, step 41 where the 

in step 21 where the VOP time offsets are decoded and bitstream encoder starts by initializing the local rime base 

passed to the decoders for synchronization. Then the mul- 35 register to the initial value of the time code. The same time 

tiplexed bitstream is examined m step 22 to see if the code value is encoded into the bitstream. At the start of the 

modulo time base is found. If the modulo time base is found encoding of the next I- VOP the time code comparator, step 

then the process is passed to step 23 where the modulo time 42, compares the presentation time of the I- VOP against the 

base is inserted into all the video object bitstreams. The local time base register. The result is passed to the moduio 
processing then continues with step 24 where the next video 40 time base encoder, step 43. The modulo time base encoder 

object is examined and inserted into the appropriate video will insert into the bitstream the required number of "1" 

object bitstream. Finally the multiplexed bitstream is exam- equivalent to the number of modulo time base increment that 

ined again to see if there is any more video object to has elapsed. This is then followed by the symbol "O" to 

de-multiplex. If there is, then the process is passed to step 22 signal the end of the modulo time base code. The local time 
again. Otherwise the process ends. 45 base register is updated to the current modulo time base. The 

& Recovering the time base is described. processing then proceed to the VOP time base increment 

The embodiment of the time base recovery is shown in encoder, step 44, where the remainder of the presentation 

FIG. 10. In recovering the local time base, the process time code of the I- VOP is encoded, 

begins in step 31 where the local time based is initialized The process then repeats with the next encoded video 
taking into account of the VOP time offset decoded by the 50 object plane which is the P-VOP. The time code comparator, 

de-multiplexer. Then the process is passed to step 32 which step 42, compares the presentation time of the P-VOP 

checlcs the bitstream to determine if the modulo time base is against the local time base register. The result is passed to 

decoded. If the modulo time base is decoded then the the modulo time base encoder, step 43. Hie modulo time 

process is passed to step 33 where the local time base is base encoder will insert the required number of M 1 M equiva- 
incremented by the modulo time base increment. The pro- 55 lent to the number of modulo time base increment that has 

cess is then p^ed to step 37. If the modulo time base is not elapsed. This is then followed by the symbol "0" to signal 

decoded then the process is passed to step 34 where the the end of the modulo time base code. The B-VOP time base 

video object is examine to determine if it is a B-video object register is set to the value of the local time base register and 

or not If it is then the process is passed to step 35 where the the local time base register is updated to the current modulo 
B-video object decoding time base is calculated based on 60 time base. The processing then proceeds to the VOP time 

equation (6). The process is then passed to step 37. If 'the base increment encoder, step 44, where the remainder of the 

result of step 34 is not a B-video object then the process is presentation time code of the P-VOP is encoded, 

passed to step 36 where the decoding time base is calculated The process then repeats with the next encoded video 

based on equation (5). The process then proceeds to step 37. object plane which is the B-VOP. The time code comparator, 
In step 37, the bitstream is examine to see if there is more 65 step 42, compares the presentation time of the B-VOP 

video objects to decode. If there is then the process is passed against the B-VOP time base register. The result is passed to 

to step 32 again. Otherwise the processing ends. the modulo time base encoder, step 43. The modulo time 
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base encoder will insert the required number of "1" equiva- in the top row in FIG. 13, the compressed video data VOPs 
lent to the number of modulo time base increment that has are aligned in the display order, II, Bl, B2 t PI, B3, P3 with 
elapsed This is then followed by the symbol "0" to signal a GOP (group of pictures) header inserted at the beginning 
the end of the modulo time base code. Both the B-VOP time of group of VOPs. While being displayed, the local time, at 
base renter and me local time base register are not changed 5 which the display is effected, is measured for each VOP 
after the processing of the B-VOP. The processing then using the local time clock. For example, the first VOP 
proceed to the VOP time base increment encoder, step 44, (Il-VOP) is displayed at 1 hour 23 minute 45 second 350 
where the remainder of the presentation time code of the rnillisecond (1:23:45:350) counted from the very beginning 
B-VOP is encoded of the video data; the second VOP (Bl-VOP) is displayed at 

The local time base register is reset at the next I-VOP 10 i:23:45:750; the third VOP (B2-VOP) is displayed at 
which marks the begirming of the next group of VOP. 1:23:46:150; and so on. 

Implementation of the Bitstream Decoder is described. For encoding the VOPs, it is necessary to insert the 

FIG. 12 illustrates the block diagram for the implemen- display time data to each VOP. The insertion of the full time 
tation of the decoder for the modulo time base and VOP time data including hour, minute, second and rnillisecond occu- 
increment to recover the presentation time stamp. As in the 15 pies a certain data space in the header portion of each VOP. 
implementation of the encoder the example in FIG. 3B is The aim of me present invention is to reduce such a data 
used. The decoding order is the same as the encoding order space and to simplify the time data to be inserted in each 
where the I-VOP is decoded followed by the P-VOP before VOP. 

the B-VOP. This is explained in the paragraphs below. Each of the VOPs shown in the first row in FIG. 13 is 

The processing begins in the initializer, step 51, where the 20 stored with the display time data of millisecond at me VOP 
local time base register is set to the value of the time code time increment area. Each of the VOPs in the first row is also 
decoded from the bitstream. The processing then proceeds stored temporarily with the display time data of hour, minute 
the modulo time base decoder, step 52, where the modulo and second. The GOP header is stored with the display data 
time base increment is decoded. The total number of modulo of hour, minute and second for the first VOP (II- VOP). 
time base increment decoded is given by the number of "1" 25 As shown in the second row in FIG. 13, the VOPs are 
decoded before the symbol "CT Next the VOP time base , delayed by a predetennined time using a buffer (not shown), 
increment is decoded in the VOP time base increment, step When the VOPs are produced from the buffer, the order of 
53. In the time base calculator, step 54, the presentation time the VOPs are changed according to the bi-directional pre- 
of the I-VOP is recovered. The total decoded modulo time t diction rule such that the bi-directional VOP, i.e., B-VOP, 
base increment value is added to the local time base register. 30 should be positioned after the P-VOP to which the B-VOP 
The VOP time base increment is then added to the local time references. Thus, the VOPs are aligned in the order of II, PI, 
base register to obtain the presentation time of the I-VOP. Bl, B2, P2, B3. 

The process then goes to the video object decoder where the As shown in the third row in HO. 13, at time Tl, i.e., 
video object is decoded. . when the GOP header is being encoded, the local time base 

For the P-VOP, the process repeats in the modulo time *35 register stores the hour, minute, second data as stored in the 
base decoder, step 52, where the modulo time base incre- * GOP header. In the example' shown in FIG. 13, the local time 
ment is decoded. The total number of modulo time base ■ base register stores 1:23:45. Then, before time T2, the 
increment decoded is given by the number of "1" decoded ; bitstream data for the GOP header is obtained with the hour, 
before the symbol w O" Next the VOP time base increment \ minute, second data being inserted as shown at the bottom 
is decoded in the VOP time base increment, step 53, In the 40 row in FIG. 13. 

time base calculator, step 54, the presentation time of the * Then, at time T2, the first VOP (Il-VOP) is taken up. The 
P-VOP is recovered. The B-VOP modulo time base register time code comparator compares the time (hour, minute, 
is set to the value in the local time base register. The total second) stored in the local time base register with the time 
decoded modulo time base increment value is added to the (hour, minute, second) temporarily stored in the first VOP 
local time base register. The VOP time base increment is '45 (Il-VOP). According to the example, the result of the 
then added to the local time base register to obtain the comparison is the same. Thus, the comparator produces "0** 
presentation time of the P-VOP. The process then goes to the indicating that the first VOP (Il-VOP) occurs in the same 
video object decoder where the video object is decoded. second as the second being held in the local time base 
For the B-VOP, the process repeats in the modulo time register. Hie result "0" as produced from the comparator is 
base decoder, step 52, where the modulo time base incre- 50 added to the first VOP (Il-VOP) at the modulo time base 
ment is decoded The total number of modulo time base area. At the same time, the hour, minute and second data 
increment decoded is given by the number of "1" decoded temporarily stored in the first VOP (Il-VOP) is elimmated. 
before the symbol "O". Next the VOP time base increment Thus, before time T3, the bitstream data for the first VOP 
is decoded in the VOP time base increment, step 53. In the (U-VOP) is obtained with "0" inserted in the modulo time 
time base calculator, step 54, the presentation time of the 55 base area and "350" inserted in the VOP time increment 
B-VOP is recovered The total decoded modulo time base area. 

increment value and the VOP time base increment is added Then, at time T3, the second VOP (Pl-VOP) is taken up. 
to the B-VO? time base register to obtain the presentation The time code comparator compares the time (hour, minute, 
time of the B-VOP. Both the B-VOP time base register and second) stored in the local time base register with the time 
the local time base register are left unchanged The process 60 (hour, rninute, second) temporarily stored in the second VOP 
then goes to the video object decoder where the video object (Pl-VOP). According to the example, the result of the 
is decoded. comparison is such that the time temporarily stored in the 

The local time base register is reset at the next I-VOP second VOP (Pl-VOP) is one second greater than the time 
which marks the beginning of the next group of VOR stored in local time base register. Thus, the comparator 

Specific example is described, 65 produces "10" mdicating that the second VOP (Pl-VOP) 

Referring to FIG. 13, an example of the steps for encoding occurs in the next one second to the second being held m the 
the compressed data into bitstream data is shown. As shown local time base register. If the second VOP (Pl-VOP) occurs 
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in the next— next second to the second being held in the 
iocai time base register, the comparator will produce "110", 

After time T3 } the B-VOP time base register is set to the 
dme equal to the time carried in the local time base register 
immediately before time T3. In the example, the B-VOP 
time base register is set to 1:23:45. Also, after time T3, the 
local time base register is incremented to the time equal to 
the time temporarily stored in the second VOP (Pl-VOP). 
Thus, in the example, the local dme base register is incre- 
mented to 1:23:46. 

The result "10" as produced from the comparator is added 
to the second VOP (Pl-VOP) at the modulo time base area. 
At the same dme, the hour, minute and second data tempo- 
rarily stored in the second VOP (Pl-VOP) is eliminated. 
Thus, before time T4, the bitstream data for the second VOP 
(Pl-VOP) is obtained with "10" inserted in the modulo time 
base area and "550" inserted in the VOP time increment 
area. 

Then, at time T4 T the third VOP (Bl-VOP) is taken up. 
The time code comparator compares the time (hour, minute, 
second) stored in the B-VOP time base register with the time 
(hour, rninute, second) temporarily stored in the third VOP 
(Bl-VOP). According to the example, the result of the 
comparison is the same. Thus, the comparator produces "0" 
indicating that the third VOP (Bl-VOP) occurs in the same 
second as the second being held in the B-VOP time base 
register. The result "0" as produced from the comparator is 
added to the third VOP (Bl-VOP) at the modulo time base 
area. At the same time, the hour, minute and second data 
temporarily stored in the third VOP (II- VOP) is eliminated. 
Thus, before time T5, the bitstream data for the third VOP 
(Bl-VOP) is obtained with "0** inserted in the modulo time 
base area and "750" inserted in the VOP time increment 
area. 

Then, at time T5, the fourth VOP (B2-VOP) is taken up. 
The time code comparator compares the time (hour, rninute, 
second) stored in the B-VOP rime base register with the dme 
(hour, minute, second) temporarily stored in the fourth VOP 
(B2-VOP). According to the example, the result of; the 
comparison is such that the time temporarily stored in the 
fourth VOP (B2-VOP) is one second greater than the time 
stored in B-VOP time base register. Thus, the comparator 
produces "10" indicating that the fourth VOP (B2-VOP) 
■occurs in the next one second to the second being held in the 
B-VOP time base register. 

During the processing of the B type VOP, neither the local 
dme base register nor me B-VOP time base register is 
incremented regardless of whatever result the comparator 
produces. 

The result "10" as produced from the comparator is added 
to the fourth VOP (B2-VOP) at the modulo time base area. 
At the same time, the hour, minute and second data tempo- 
rarily stored in the fourth VOP (B2-VOP) is eliminated. 
Thus, before time T6 y the bitstream data for the fourth VOP 
(B2-VOP) is obtained with "10" inserted in the modulo time 
base area and "150" inserted in the VOP time increment 
area. 

Then, at time T6, the firth VOP (P2-VOP) is taken up. The 
time code comparator compares the time (hour, minute, 
second) stored in the local time base register with the time 
(hour, minute, second) temporarily stored in the fifth VOP 
(P2-VOP). According to the example, the result of the 
comparison is such that the time temporarily stored in the 
fifth VOP (P2-VOP) is one second greater than the time 
stored in local rime base register. Thus, the comparator 
produces "10" indicating that the fifth VOP (P2-VOP) 
occurs in the next one second to the second being held in the 
local time base register. 
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After time T6, the B-VOP dme base register is incre- 
mented to the time equal to the time carried in the local time 
base register immediately before time T6. in the example, 
the B-VOP time base register is incremented to 1:23:46. 
5 Also, after time T6, the local time base register is incre- 
mented to the time equal to the time temporarily stored in the 
fifth VOP (P2-VOP). Thus, in the example, the local time 
base register is incremented to 1:23:47. 
The result " 10" as produced from the comparator is added 
10 to the fifth VOP (P2-V0P) at the modulo dme base area. At 
the same time, the hour, minute and second data temporarily 
stored in the fifth VOP (P2-VOP) is eliminated. Thus, before 
dme T4, the bitstream data for the fifth VOP (F2-VOP) is 
obtained with "10" inserted in the modulo time base area and 
15 "350" inserted in the VOP time increment area. 

Thereafter, a similar operation is carried out for forming 
bitstream data for the following VOPs. 

For decoding the bitstream data, an operation opposite to 
the above is carried out First, the time (hour, minute, 
20 second) carried in the GOP header is read. The read time is 
stored in the local time base register. 

Upon receipt of the I type or P type VOP, i.e., other then 
the B type VOP, the data stored in the modulo time base area 
is read. If the read data is "0", i.e., without any is in front of 
25 0, no change Is made in the local time base register, and also 
r no change is made in the B-VOP time base register. If the 
read data is "10", the time stored in the local time base 
register is incremented by one second. If the read data is 
t "110*\ the time stored in the local dme base register is 
30 * incremented by two seconds. In this manner, the number of 
seconds which should be incremented is determined by the 
number of is inserted in front of 0. Also, when the read data 
is "10" or "110", the B-VOP time base register, which is a 
memory, copies the time the local time base register had 
35 ■ carried just before the most recent increment Then, the time 
* (hour, minute, second) carried in the local time base register 
■ is combined with the time (millisecond) carried in the VOP 
: time increment area to establish a specific time at which the 
I type or P type VOP should occur. 
40* Upon receipt of the B type VOP, the data stored in the 
. modulo time base area is read. If the read data is "0", the 
time (hour, minute, second) carried in the B-VOP VOP time 
base register is combined with the time (millisecond) carried 
in the VOP time increment area to establish a specific dme 
45 at which the B type VOP should occur. If the read data is 
"10", the time (hour, minute, second) carried in the B-VOP 
time base register is added one second, and then the added 
result time is combined with the dme (millisecond) carried 
in the VOP time increment area to establish a specific time 
50 at which the B type VOP should occur. If the read data is 
"110", the time (hour, minute, second) carried in the B-VOP 
time base register is added two seconds, and then the added 
result time is combined with the time (millisecond) carried 
in the VOP time increment area to establish a specific time 
55 at which the B type VOP should occur* 

The effects of this invention is to enable video object 
planes encoded by different encoders to be multiplexed. It 
will also facilitate object based bitstream manipulation of 
compressed data that come from .different sources to create 
60 new bitstreams. It provides a method of synchronization for 
audio visual objects. 

The invention being thus described, it will be obvious that 
the same may be varied in many ways. Such variations are 
not to be regarded as a departure from the spirit and scope 
65 of the invention, and all such modifications as would be 
obvious to one skilled in the art are intended to be included 
within the scope of the following claims. 



